Ambari Custom Service Learning

Integrate Kafka-Manager to Ambari

Deposit address:

CD/var/lib/ambari - server/resources/sports/HDP / 2.5 / services

The final file structure

─ Configuration │ ├─ Kafka_Manager ├── MetaInfo.xml ├─ Package ├─ Archive.zip │ ├─ Scripts │ sigma ─ sigma ─ sigma ─ sigma ─ sigma ─ sigma ─ sigma ─ sigma ─ sigma

1. Write the metainfo. XML

<? The XML version = "1.0"? BB0 < MetaInfo > </ SchemaTick2 2.0</ SchemaTick3 <services> <service> <! -- Internal name for service (must be unique) --> <name>KAFKA_MANAGER</name> <! -- display name in Ambari UI --> <displayName>KAFKA_MANAGER</displayName> <! -- Description of service - will be displayed when user clicks add service --> <comment>the kafka's manager</comment> <! -- Version of service --> < Version >1.3.2.1</ Version > < Components -- In this case, there is only one master component --> <component> <name>KAKFA_MASTER</name> <displayName>KAFKA_MANAGER</displayName> <category>MASTER</category> <! -- how many of these components are allowed in a cluster --> <cardinality>1</cardinality> <! -- reference to (and details of) what script is to be used to install/stop/start/config the service --> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> </components> <! -- Package to be installed, >< ospecify >< osFamily>any</osFamily> </packages> </ ospecify ></ ospecify ></ ospecify ><! > <configuration-dependencies> <config-type>kafka-manager-env</config-type> </configuration-dependencies> <! - after modify configuration, whether to need to restart the service - > < restartRequiredAfterChange > false < / restartRequiredAfterChange > <! > < QuickLinksConfigurations > < QuickLinksConfiguration > <fileName> QuickLinks. Json </fileName> <default>true</default> </quickLinksConfiguration> </quickLinksConfigurations> </service> </services> </metainfo>

2. Write the script mast.py

#! /usr/bin/env python from resource_management import * class Master(Script): Def install(self, env) def install(self, env) Print 'Install the KAFKA Master' print 'Install the KAFKA Master' print 'Install the KAFKA Master' print 'Install the KAFKA Master' Execute('ps -ef | grep kafka-manager | grep -v grep | awk \'{print $2}\' | xargs kill -9') Execute('rm -f /opt/kafka-manager-1.3.2.1/RUNNING_PID') print 'def start(self, env): The Execute (' nohup/opt/kafka - manager - 1.3.2.1 / bin/kafka - manager - Dconfig. The file = / opt/kafka - manager - 1.3.2.1 / conf/application. The conf - Dhttp. Port = 6888 > /opt/kafka-manager-1.3.2.1/ kafka-manager-1.3.2.1/kafka-manager.log 2 >&1&') # def status(self, env): /opt/kafka-manager-1.3.2.1/kafka-manager.log 2 >&1&' Check_process_status ('/opt/kafka-manager-1.3.2.1/RUNNING_PID') # config file def (self, env) print 'Configure the KAFKA Master'; if __name__ == "__main__": Master().execute()

3. Write QuickLinks. JSON file

{
  "name": "default",
  "description": "default quick links configuration",
  "configuration": {
    "protocol":
    {
      "type":"https",
      "checks":[
        {
          "property":"ssl_enable",
          "desired":"true",
          "site":"kafka-manager-env"
        }
      ]
    },

    "links": [
      {
        "name": "manager_ui",
        "label": "Kafka Manager",
        "requires_user_name": "false",
        "component_name": "KAKFA_MASTER",
        "url":"%@://%@:%@/",
        "port":{
          "http_property": "kafka.server.port",
          "http_default_port": "6888",
          "https_property": "kafka.server.port",
          "https_default_port": "6888",
          "regex": "^(\\d+)$",
          "site": "kafka-manager-env"
        }
      }
    ]
  }
}

4. Write configuration files

<? The XML version = "1.0"? > <? xml-stylesheet type="text/xsl" href="configuration.xsl"? > <! -- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You are under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> <configuration> <property> <name>kafka.server.port</name> <value>6888</value> <display-name>HTTP Post</display-name> <description>Webserver listens on this port</description> </property> </configuration>

5. Restart ambari – server

ambari-server restart

6. Add Service to Web

You can see that KAFKA_MANAGER is in our list of services

Click Next to add Kafka_Manager Service

Try to see if Stop, Restart All, and QuickLinks work properly

There is no problem with verification

Integrated Java programs

Java program file structure, lib package too much not listed, the relevant JAR package all into the lib package, temporarily only support XML file format

─ config │ ├─ Application. YML │ ├── ─ java-configur.xml ├── Bass Exercises ─ Log Exercises ── Stdio. Log Exercises ── Classes javaControl.py

JavaControl. Py is a Python script that you can use to start and close Java programs. This is a Python script that you can use to start and close Java programs

#! /usr/bin/env python # -*- coding: UTF-8 -* -import commands import OS import sys import subprocess _programName = "springboot # main Class _pythonName = sys.argv[0] # workplace = os.getcwd() + os.sep # < pidFileName = "pid.txt" > < / pidFileName = "pid.txt" > < / pidFileName = "pid.txt" > Print "Please use %s help to see how to use the script" % _pythonName # Print: %s start % _pythonName print: %s stop % _pythonName print: %s stop % _pythonName print: %s stop % _pythonName %s restart" % _pythonName # pid" Return os.path.exists(_Workplace + _PidFileName) # def startProgram(): if pidisExist (): print GetProgramHelpIntroduce () else: print os.system( "exec -a %s java -Xmx1024m -cp config/:lib/* %s >> log/stdio.log 2>&1 & echo $! >%S &" % (_programName, _programClassName, _Workplace + _PidFileName)) print "Program is running successfully! Def stopProgram(): if not pidIsExist(): print os.system("rm -f %s" % _workPlace + _pidFileName) getProgramHelpIntroduce() else: pid = int(commands.getoutput(" cat %s " % (_workPlace + _pidFileName))) os.system("kill %s" % pid) os.system("rm -f %s" % _Workplace + _PIDFileName) print "Program closed successfully!" if __name__ == "__main__": if len(sys.argv) == 2: args = sys.argv[1] else: args = raw_input("Please Enter Your Command: ") logPath = _workPlace + "log" if not os.path.exists(logPath): os.mkdir(logPath) if args == "start": startProgram() elif args == "stop": stopProgram() elif args == "restart": stopProgram() startProgram() elif args == "help": getProgramHelpCommand() else: getProgramHelpIntroduce()

Next we do the Ambari integration with Java, where the java-config is a file structure copied from the Java program above

JAVA_TEST/ configuration │ ├─ sigma ── sigma ── metaInfo ── package sigma ─ scripts │ sigma ─ master.py │ sigma ── sigma params.py

metainfo.xml

<? The XML version = "1.0"? BB0 < MetaInfo > </ SchemaTick2 2.0</ SchemaTick3 <services> <service> <! -- Internal name for service (must be unique) --> <name>JAVA_TEST</name> <! -- display name in Ambari UI --> <displayName>JAVA_TEST</displayName> <! -- Description of service - will be displayed when user clicks add service --> <comment>a java application test</comment> <! -- Version of service --> < Version >1.0.0.test</ Version > < Components > <! -- In this case, there is only one master component --> <component> <name>JAVA_MASTER</name> <displayName>JAVA_MAIN</displayName> <category>MASTER</category> <! -- how many of these components are allowed in a cluster --> <cardinality>1</cardinality> <! -- reference to (and details of) what script is to be used to install/stop/start/config the service --> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> </components> <osSpecifics> <osSpecific> <osFamily>any</osFamily> <packages></packages> </osSpecific> </osSpecifics> <configuration-dependencies> <config-type>java-config</config-type> </configuration-dependencies> <restartRequiredAfterChange>false</restartRequiredAfterChange> <quickLinksConfigurations> <quickLinksConfiguration> <fileName>quicklinks.json</fileName> <default>true</default> </quickLinksConfiguration> </quickLinksConfigurations> </service> </services> </metainfo>

params.py

#! /usr/bin/env python # -*- coding: utf-8 -*- from resource_management.libraries.script.script import Script config = Script.get_config() java_config = config['configurations']['java-config']

master.py

#! /usr/bin/env python from resource_management import * import os class Master(Script): def install(self, env): self.install_packages(env) #To stop the service, use the linux service stop command and pipe output to log file def stop(self, env): os.chdir('/usr/local/share/applications/javaSpringboot') Execute('./javaControl.py stop') #To start the service, use the linux service start command and pipe output to log file def start(self, env): import params self.configure(env) os.chdir('/usr/local/share/applications/javaSpringboot') Execute('./javaControl.py start') #To get status of the, use the linux service status command def status(self, env): check_process_status('/usr/local/share/applications/javaSpringboot/pid.txt') def configure(self, env): import params env.set_params(params) XmlConfig("java-config", conf_dir="/usr/local/share/applications/javaSpringboot/config", configurations=params.java_config, owner="root", group="root", mode=0644 ) if __name__ == "__main__": Master().execute()

Integrated tomcat

ambari tomcat

Pits stepped in the integration

1. The ambari-server restart will report an error database check failed

The 2017-10-16 11:21:52 INFO - 736 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * Check the database started * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 2017-10-16 11:21:56,029 Info - Checking for configs not mapped to any cluster 2017-10-16 11:21:56,048 Info - Checking For configs selected more than once 2017-10-16 11:21:56,050 info-checking for hosts without state 2017-10-16 Info - Checking Host Component States Count Equals Host Component Dummy States count 2017-10-16 2017-10-16 11:21:58,035 error-service (s): 2017-10-16 11:21:58,035 error-service (s): KAFKA_MANAGER, from cluster cx has no config(s) in serviceconfig table!

Ambari Server Configuration Modification Process

2. QuickLinks cannot be displayed

Check that metaInfo.xml is configured correctly for quicklinks, and check that the quickLinks. JSON file is correct

Ambari does not report errors, just does not display QuickLinks

Links to resources

  • Ambari Custom Service Steps Description (1, 2, 3, 4) CSDN
  • Ambari custom service documents
  • Ambari custom service related scripts github source code
  • Introduction to Python 2.7 by Liao Xuefeng
  • Ambari service configuration related
  • Ambari Custom Artifact