“This article has participated in the good article call order activity, click to see: back end, big front end double track submission, 20,000 yuan prize pool for you to challenge!”
1:DevOps
Dafang’s operational wiki is developed by enabling operation and maintenance tools. In the actual development process, operation and maintenance rarely participate in the launch and daily management of a project. In this way, the requirements for development will rise to a higher level. When encountering related problems online, it is necessary to immediately understand the solution of related problems and the significance of the execution of related commands. Thus fast and accurate positioning line problem. Next, let’s take a look at the common commands used in daily work in case of emergency.
2: common commands of components
1: common command for online problems
## find the thread with the highest CPU usage in the process top-hp PID ##top thread printf"%x\n" 21742# # it is recommended to use root jstack pid | grep 54 eeCopy the code-type f-size + 800m ## Find the large folder du -hm --max-depth=2 | sort -nr | head -12## check nginx request maximum IP cat/ * * * /nginx/logs/access.log|awk '{print $1,$2}'sort|uniq -c|sort -rn -k1|head -3Nohup java-jar publish_online.jar --server.port=8106& >/opt/Data /xxx_8106.log & ## find /opt/shell directory and subdirectory containing redis keyword grep -ril"redis" /opt/shell/ *Copy the code2: common Hadoop commands
HDFS dfsadmin -report HDFS dfsadmin -report HDFS dfsadmin -report HDFS dfsadmin -report hadoop DFFS -ls /user/hive/ ## HDFS deleteSnapshot HDFS DFFS -deletesnapshot /user/hive/hive-snapshot-20210509Download data block hadoop fs - get/user/hive/warehouse/ods db/table_name/part -00021-d20b1d30-dCE1-4a38-A0C1-2b4ef68b7da8-c000.snappy. Orc./ # Download the Yarn log Yarn logs-ApplicationID # HDFS FSCK /user/events -openForwrite # HDFS FSCK /user/events -openForwrite # file lease HDFS debug recoverLease - path/user/hive/warehouse/app. Db/test_order_sample1 / order_sample - retries3Oev -i edits_0000000001177152591-0000000001177162768 -o edits_0000000001177152591-0000000001177162768.Oev -i edits_0000000001177152591-0000000001177162768.xml -o edits_0000000001177152591-0000000001177162768-p binary ## Check bad HDFS data blocks hadoop FSCK/user-files-block-locations ## Delete bad HDFS blocks hadoop FSCK -rm /user/spark/sparkstreamingcheckpoint/checkpoint-1517487380000# # file statistics data block under HDFS DFS - count/user/hive/warehouse/DWS. DbDistcp -d dfs.replication=2 HDFS ://src-hadoop-address:9000/data/ TMP / hdfs://dst-hadoop-address:9000/data/hfile/ >> /home/data/logs/hadoop-${date}.log 2>&1Copy the code3: common kafka command
# #518400000Kafka -configs --zookeeper zkhost:2181 --alter --entity-name topic_name --entity-type topics --add-config retention.ms=518400000Configs --zookeeper zkhost --zookeeper zkhost2181 --alter --entity-name topic_name --entity-type topics --add-config retention.ms=-1## reset kafka-consumer-groups --bootstrap-server kafkahost:9092 --group event_tag --reset-offsets --topic topic_name --execute --to-datetime 2021-04-23T08:20:00000.## reset kakfa's offset kafka-consumer-groups --bootstrap-server kafkahost:9092 --group event_tag --topic topic_name --execute --reset-offsets --to-offset 5425016603## Check all topic kafka-topics --list --zookeeper zkhost:2181## Delete topic kafka-topics --delete --zookeeper zkhost:2181Kafka -topics --alter --zookeeper zkhost:2181 --topic topic_name --partitions 10 --replication-factor 2## reset the original kafka-consumer-groups --bootstrap-server kafkahost:9092--group group_name --reset-offsets -- toptopic_name -- to-execute ## --bootstrap-server kafkahost:9092 --group group_name --reset-offsets --topic topic_name --to-latest --execute
Copy the code4: common elasticsearch command
POST /ex_index/_mapping for elasticSearch index"properties" : {
"last_buy_time": {
"type" : "date"."format" : "epoch_millis"}}} ## elasticsearch ## elasticsearch ## elasticsearch ## curl -s -H"Content-Type: application/json" -XGET ip:9200/_sql -d "select sum(install_device_count) as total from table_name where hour=9 and staticdate='2021-03-09' and channel_number='oppo'" |jq '.aggregations.total.value'DELETE /es_index/_doc/id. DELETE /es_index/_doc/id"Content-Type:application/json" http://ip:9200/ex_index/_settings -d '{ "index" : { "number_of_replicas" : "0" }}'## update os_cache 'segment' to 'curl' -xput -h"Content-Type:application/json" http://ip:9200/ex_index/_settings -d '{ "index" : { "refresh_interval" : "60s" }}'## ElasticSearch write translog async curl -xput -h"Content-Type:application/json" http://ip:9200/ex_index/_settings -d '{ "index" : { "translog": { "durability" : "async" }}}'## ElasticSearch ES_index Mapping GET ES_index /_mapping GET ES_index /_mapping POST ex_index/_search {"query": {"range" : {
"staticdate" : {
"from" : "2020-12-06"."to" : "2020-12-06"."include_lower" : true."include_upper" : true."boost" : 1.0}}},"size":20} ## load POST _bulk {"create" : { "_index" : "bulk_index"."_id" : "1"}} {"Job" : "Linux Ops" }
{ "delete" : { "_index" : "bulk_index"."_id" : "2"}} {"update" : { "_id": "1"."_index" : "bulk_index"}} {"doc" : {"name" : "xuweiliang"}} {"index" : {"_index" : "bulk_index"."_id" : "1"}} {"name" : "xuweiliang" }
{ "create" : { "_index" : "bulk_index"."_id" : "2"}} {"name" : "xuweiliang" , "Age" : 25 }
{ "delete" : { "_index" : "bulk_index"."_id" : "2"}} ## POST es_score_test/_search {"query": {
"function_score": {
"query": {
"match": {
"category_id":1}},"script_score": {
"script": "return doc ['second_nature_id'].value == 20 ? 1.1:0.5"}}}}Copy the code5: hbase common commands
# # hbase bulk import data in hbase. Org. Apache hadoop, hbase. Graphs. LoadIncrementalHFiles/data/hfile / ${hbaseFileDir}"${tableName}" >> /home/bigdata/logs/hbase-${date}.log 2> &1## Hive Create hbase appearance create table`dws.table_name`(
`rowkey` string COMMENT 'rowkey'.`userid` string COMMENT 'userid'.`deviceid` string COMMENT 'deviceid' ,
`last_login_time` string COMMENT 'last_login_time'.`use_count` INT COMMENT 'use_count'.`reserve1` string COMMENT 'Reserved field 1'.`reserve2` string COMMENT 'Reserved field 2'
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH
SERDEPROPERTIES ("hbase.columns.mapping"=
":key, cf:userid, cf:deviceid, cf:last_login_time, cf:use_count, cf:reserve1, cf:reserve2) TBLPROPERTIES ("hbase.table.name"="table_name"); Create hbase table, specify compression mode, Create 'ads:table_name', {NAME => 'cf', COMPRESSION => 'SNAPPY', VERSIONS => 3},SPLITS => ['10','15','20','25','30','35','40','45','50']Copy the code6: flink common commands
nohup flink run -m yarn-cluster -yjm 1024m -ytm 2048m -ynm SyncStream -ys 1 -yqu queueName -c com.xxx.SyncStream /data/sync-datapretreatment.jar $1 > /data/SyncStream.log 2> &1 &
Copy the code7: common spark command
Set spark.yarn.queue=root.default; set spark.app.name=merge; # # dynamic resource set spark. DynamicAllocation. MaxExecutors =10;
set spark.dynamicAllocation.enabled=false; # # partition set set hive. The exec. Max. Dynamic. Partitions =10000;
set hive.exec.max.dynamic.partitions.pernode=10000;
set hive.exec.dynamic.partition=true; set hive.exec.dynamic.partition.mode=nonstrict; ## Execute the resource set spark.executor.instances=5;
set spark.executor.memory=6g;
set spark.driver.memory=3g;
set spark.executor.cores=4; set spark.memory.offHeap.size=2g; # file merging adaptive set spark. SQL. The adaptive. Shuffle. TargetPostShuffleInputSize =134217728;
set spark.sql.adaptive.enabled=true; Nohup spark-submit --master yarn --deploy-mode cluster --driver-memory 3g /opt/product/sparkjob/datacalculate.jar >>/opt/product/sparkjob/log/datacalculate.log 2> &1 &
Copy the code8: Kylin common commands (small file cleaning)
## Check which resources can be cleaned, (--delete false) will not delete any files:export KYLIN_HOME=/path/to/kylin_home
${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete falseYou can spot check the relevant documents to check whether they are still in use. After confirming that there is no problem, add'- delete true'Option to clean up. ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete trueAfter the operation is complete, the intermediate tables in Hive, the intermediate files in HDFS, and HTables in HBase are removed. If you want to delete all resources; You can add'- the force true'Options: ${KYLIN_HOME} / bin/kylin. Sh org. Apache. Kylin. Tool. StorageCleanupJob -- forcetrue --delete trueAfter the operation is complete, all intermediate tables in Hive, all intermediate files in HDFS, and HTables in HBase are removed.Copy the code9: Common clickhouse commands
Clickhouse-client-m --port9002 -u defaultAlter table table_name on cluster cluser_name modify column user_name Nullable(String); Set TTL ALTER TABLE for local TABLEdefault.tablename on cluster cluser_name modify TTL staticdate + INTERVAL 1Year; ## Delete cluster data from clickhouse-client --port9002 -u default --password xxx -h 127.0. 01. --query "alter table default.table_name on cluster cluser_name delete where staticdate >= '${1}'"CREATE TABLE CREATE TABLE CREATE TABLEdefault.tablename on cluster cluser_name
(
distinct_id String,
event String,
staticdate Date,
project String,
event_date_time Nullable(DateTime)
)
ENGINE = MergeTree
PARTITION BY formatDateTime(staticdate, '%Y-%m')
ORDER BY (event, staticdate,project, javaHash(distinct_id))
SETTINGS index_granularity = 8192
,storage_policy = 'hdd_in_order'; CREATE TABLE CREATE TABLE CREATE TABLEdefault.tablename_all on cluster cluser_name
(
distinct_id String,
event String,
staticdate Date,
project String,
event_date_time Nullable(DateTime)
) ENGINE = Distributed('cluser_name'.'default', local_table_name, rand());
Copy the code10: common redis commands
##redis to clear the cache (with caution) redis-cli -p6379 -a 'password' -n 0 keys "*" | xargs redis-cli -p 6379 -a 'delete key starting with XXX' -n 0Del ## Query bigkey redis-cli -h IP -p port -a password --bigkeys -i0.01
Copy the code3:
Memorizing common online commands only deeps your understanding of the components, and solving problems online requires a deeper understanding of the business scenarios involved. By combining an understanding of the business scenario with a technical solution, you will be able to handle the relevant problems with ease. Temporary arrangement, there may be some omissions, the follow-up continuous supplement.