juju debug hacks (by quqi99)

Copyright notice ： You can reprint at will , When reprinting, please make sure to indicate the original source of the article, the author's information and this copyright notice in the form of hyperlinks ( author ： Zhang hua Published in ：2021-07-22)

debug juju-controller

juju ssh -m controller 0
juju_engine_report | pastebinit -f yaml
eg:
instance-poller:
                inputs:
                - api-caller
                - environ-tracker
                - clock
                - valid-credential-flag
                - migration-fortress
                - migration-inactive-flag
                start-count: 19847
                state: starting

Above said instance-poller There is a problem . instance-poller be used for juju-controller On a regular basis from maas-api Rotation maas machine/instance The data of . As possible below and spaces relevant .

juju reload-spaces --debug --verbose --show-log
ERROR could not reload spaces: unexpected: Get "http://10.91.42.24/MAAS/api/2.0/spaces/": EOF
16:16:21 DEBUG cmd supercommand.go:537 error stack:
unexpected: Get "http://10.91.42.24/MAAS/api/2.0/spaces/": EOF

Check the maas End sum juju Terminal spaces Whether the information is consistent , If not, try restarting juju-controller try .

maas admin spaces read
juju spaces

#juju spaces
ceph-access-space 5 10.91.246.0/23
ceph-replica-space 4 10.91.250.0/23
external-space 3 10.91.248.0/23
internal-space 2 10.91.244.0/23
oam-space 1 10.91.42.0/23
#cat maas-admin-space-read.txt | jq '.[] | {name:.name,id:.id,url:.resource_uri,subnet:.subnets[].cidr}' -c
{"name":"oam-space","id":1,"url":"/MAAS/api/2.0/spaces/1/","subnet":"10.91.42.0/23"}
{"name":"internal-space","id":2,"url":"/MAAS/api/2.0/spaces/2/","subnet":"10.91.244.0/23"}
{"name":"external-space","id":3,"url":"/MAAS/api/2.0/spaces/3/","subnet":"10.91.248.0/23"}
{"name":"ceph-replica-space","id":4,"url":"/MAAS/api/2.0/spaces/4/","subnet":"10.91.250.0/23"}
{"name":"ceph-access-space","id":5,"url":"/MAAS/api/2.0/spaces/5/","subnet":"10.91.246.0/23"}
{"name":"undefined","id":-1,"url":"/MAAS/api/2.0/spaces/undefined/","subnet":"192.168.122.0/24"}

# why undefine space is missing, see db
juju ssh -m controller 0
mongo --sslAllowInvalidHostnames --sslAllowInvalidCertificates localhost:37017/admin --ssl -u "$(ls /var/lib/juju/agents/)" -p "$(sudo grep statepassword /var/lib/juju/agents/$(ls /var/lib/juju/agents/)/agent.conf|awk '{print $2}')" --authenticationDatabase admin
use juju;
db.spaces.find().pretty()
mongodump --port 37017 --sslAllowInvalidCertificates --ssl --authenticationDatabase admin -u "$(sudo awk '/^tag:/ { print $2 }' /var/lib/juju/agents/machine-*/agent.conf)" -p "$(sudo awk '/^statepassword:/ { print $2 }' /var/lib/juju/agents/machine-*/agent.conf)" --db juju --gzip
2)tar -czf mongodump-`date '+%F_%T'| tr -s ':' '-'`.tgz dump

No matter controller side Is there one less undefine Of space Well , At least ：

• instance-poller Visiting /MAAS/api/2.0/machines It's out EOF wrong
• reload-spaces Also came out. EOF wrong
  Both of these points indicate controller and maas Of 80/5240 The port seems to be blocked ． So next you can maas Installation on maas-cli Try again to troubleshoot network problems ． But use ’maas admin spaces read’ You can see the log of the following requests , but reload-spaces It seems that you can't see ．

... regiond: [info] 10.91.42.24 GET /MAAS/api/2.0/spaces/ HTTP/1.1 --> 200 OK (referrer: -; agent: Python-httplib2/0.9.2 (gzip))

Is the connection proxied （ juju model-config -m controller ）, Grab Test ,10.91.43.2 It's one of them controoler.

sudo tcpdump -i any src host 10.91.43.2 -w pcap-02.pcap
juju reload-spaces --debug --verbose --show-log

Continue to grab extra bags ：

tcpdump -i any dst host 10.91.42.24 -w pcap-controller.pcap on machine 10.91.43.2
tcpdump -i any src host 10.91.43.2 -w pcap-maas.pcap on machine 10.91.42.24
juju reload-spaces
maas admin spaces read

Is it haproxy There is a problem , Bypass haproxy try ：

sudo bash -c 'cat > testcloud.yaml' << EOF
clouds:
  foundations-maas:
    type: maas
    auth-types: [oauth1]
    endpoint: 
      http://10.91.42.23:5240/MAAS
EOF
juju update-cloud maas-cloud -f testcloud.yaml --client --controller new_maas_cloud_controller
juju reload-spaces --debug --verbose --show-log

Finally, because of this setting ,comment After that ( The customer added it himself )．

#defaults
# timeout connect 10s
# timeout client 30s
# timeout server 30s